Wide dynamic range (WDR) processing is becoming increasingly popular in modern-day imaging systems. Wide dynamic range describes an attribute of an imaging system that can record greater scene details, from shadows to highlights, than normal. WDR images are able to capture a much higher dynamic range and as such are useful for applications like security and automotive which are expected to generate correctly exposed images even in adverse lighting conditions. Such adverse lighting conditions might include surveillance cameras operating in the morning/evening hours when the sun is low on the horizon and facing the camera, automotive cameras when the car is emerging from a dark parking lot to a sunlit road, and indoor scenes, e.g., a dark room with a sun-lit window, where both the interior of the room as well as the scenery outside should be properly exposed. Under such extreme lighting conditions, it is desirable that the generated image have minimal areas of blow-out regions while still showing good detail in the dimly-lit portions of the image. Tone-mapping image processors can be used to address this problem.
FIG. 1 is a data-flow diagram representing a typical data path 100 for a standard (non-WDR) image processing pipeline. In FIG. 1, the raw image data consists of 12-bit data blocks, each of which represents one image pixel. All of the image processor blocks support only 12-bit data width and the image pipe is able to support a maximum of 12-bit raw pixel data at one pixel/cycle throughput. In FIG. 1, an image sensor stores sensed image data in memory 110. A read-DMA unit 120 reads the raw image data from memory 110 using a direct-memory-access operation and forwards it to a pre-processing block 130 of an image processor 125. The pre-processing block 130 performs various image processing functions and provides the pre-processed image data to the image pipe 140 of the image processor 125. The image pipe 140 is a hardware image-processing module that generates image data in a particular image data format (such as YUV format) from raw image data. The image pipe 140 also performs various image processing functions. It will be noted that while FIG. 1 is a data-flow diagram as opposed to a functional block diagram, the pre-processing block 130 and the image pipe 140 are shown to be within the dashed box 125 representing the image processor to indicate that the functional blocks that perform those functions are included within the image processor. The image processor 125 stores the processed and formatted image data generated by the image pipe 140 in memory 110. The image processor blocks, i.e., pre-processing block 130 and image pipe 140 support only 12-bit data width. Thus the image pipe 140 is able to support a maximum of 12-bit raw pixel data at one pixel/cycle throughput.
FIG. 2 is a data-flow diagram representing a typical data path 200 supporting wide-dynamic-range image processing. Certain WDR image sensors can optionally compress the pixel data before sending it over the serial interface. FIG. 2 illustrates a scenario wherein the native data width is 24 bits and the pixel data is compressed within the sensor to 12 bits before storing the pixel data in memory 210. The read-DMA unit 220 reads the compressed image data from memory 210 and forwards it to a decompanding block of an image processor 225. The decompanding block 230 decompands, i.e., decompresses, the image data to its native 24 bits. The terms decompand and decompress will be used interchangeably throughout this specification. The decompanded 24-bit data is provided to pre-processing block 230 of the image processor 225. The pre-processing block 230 performs various image processing functions and provides the pre-processed image data to a global tone-mapping stage 250. Tone mapping is a technique used to map one set of colors to another to approximate the appearance of high-dynamic-range images in a medium that has a more limited dynamic range. In global tone mapping, every pixel in the image is mapped in the same way, independent of the value of surrounding pixels in the image. The global tone mapping reduces the dynamic range of the image data, from 24 to 16 bits in the example of FIG. 2. The globally tone-mapped pixel data is then provided to a local tone-mapping stage 260. In local tone-mapping, the parameters of the mapping function vary from pixel to pixel, according to parameters of the surrounding pixels. In other words, the effect of the algorithm varies in each pixel according to the local features of the image. Thus the local tone-mapping algorithms are much more complex than the global tone-mapping functions. The local tone-mapping further reduces the dynamic range of the image data, from 16 to 12 bits in the example of FIG. 2, while still preserving details in the highlights as well as the shadow regions of the image. The result of the local tone-mapping is then provided to the image pipe 270 of the image processor 225. The decompanding block 230, pre-processing block 240, global tone-mapping block 250, local tone-mapping block 260, and image pipe 270 are all shown to be within the dashed box 225 representing the image processor to indicate that the functional blocks that perform those functions are included within the image processor. The image processor 225 then stores the processed and formatted image data in memory 210.
A potential drawback of this data flow as represented by FIG. 2 is that the pre-processing block 240 needs to be expanded to support 24-bit data. The pre-processing block 240 is a complex processing block which supports a variety of processing functions including noise filtering, defect pixel correction, DC clamp removal, lens shading correction, and auto white balance correction. It is optimal for these features to be implemented before the tone-mapping logic to achieve the desired image quality. The noise filtering and defect pixel correction directly impact the quality of the tone-mapping logic, the DC clamp removal should occur before white balance is applied (preferably before tone-mapping), and similarly the lens shading correction should occur prior to applying any non-linear operation.
Enhancing the image processing modules to support a 24-bit data path requires the logic data path to be expanded to 24 bits, which has significant logic area overhead. Similarly, the line memories corresponding to each of the image processing modules have to be expanded in order to accommodate 24-bit data. The noise filters in particular tend to have significant line memories which have a huge impact on memory area. In the example illustrated by FIG. 2, the logic area would have to be doubled to support a 24-bit data path for the pre-processing blocks. Similarly, the memory area footprint would increase by roughly 80-90% to account for the 2× memory sizes (memory area does not scale linearly with the size of the memories as there is area associated with the number of ports which doesn't change in this case). Enhancing the image processing modules to support 24 bits also has the additional burden of schedule and R&D cost associated with the development.